{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get data for RAG"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2025-01-16 23:10:00--  https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.111.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 75042 (73K) [text/plain]\n",
      "Saving to: ‘pg_essay.txt’\n",
      "\n",
      "pg_essay.txt        100%[===================>]  73.28K  --.-KB/s    in 0.01s   \n",
      "\n",
      "2025-01-16 23:10:00 (5.58 MB/s) - ‘pg_essay.txt’ saved [75042/75042]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'pg_essay.txt'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build a simple RAG query engine \n",
    "- First we chunk documents from the main document, and\n",
    "- Build in-memory Vector index from documents using LlamaIndex's `VectorStoreIndex` abstraction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core import SimpleDirectoryReader, VectorStoreIndex\n",
    "from llama_index.postprocessor.longllmlingua import LongLLMLinguaPostprocessor\n",
    "\n",
    "documents = SimpleDirectoryReader(input_files=[\"pg_essay.txt\"]).load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "index = VectorStoreIndex.from_documents(documents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "retriever = index.as_retriever(similarity_top_k=8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"What did the author do during his time at Yale?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nodes = retriever.retrieve(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[NodeWithScore(node=TextNode(id_='f01289fa-87f1-4dcd-b0cc-6f44ae51629c', embedding=None, metadata={'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='798d3d31-7d21-479b-ba63-0863ee680812', node_type='4', metadata={'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, hash='0c3c3f46cac874b495d944dfc4b920f6b68817dbbb1699ecc955d1fafb2bf87b'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='dd56b836-1549-416e-824b-88989aca584c', node_type='1', metadata={'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, hash='4eb12f7e12915ac981f2e09cb17569b178f96226b7fa04b101660e159a4e9beb'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='223342ff-9d55-457c-873d-fa78289fb250', node_type='1', metadata={}, hash='80740b560cf1f167852baeff65023df33b3142005cbdff5b9451a24c6ce99907')}, metadata_template='{key}: {value}', metadata_separator='\\n', text='I remember when my friend Robert Morris got kicked out of Cornell for writing the internet worm of 1988, I was envious that he\\'d found such a spectacular way to get out of grad school.\\n\\nThen one day in April 1990 a crack appeared in the wall. I ran into professor Cheatham and he asked if I was far enough along to graduate that June. I didn\\'t have a word of my dissertation written, but in what must have been the quickest bit of thinking in my life, I decided to take a shot at writing one in the 5 weeks or so that remained before the deadline, reusing parts of On Lisp where I could, and I was able to respond, with no perceptible delay \"Yes, I think so. I\\'ll give you something to read in a few days.\"\\n\\nI picked applications of continuations as the topic. In retrospect I should have written about macros and embedded languages. There\\'s a whole world there that\\'s barely been explored. But all I wanted was to get out of grad school, and my rapidly written dissertation sufficed, just barely.\\n\\nMeanwhile I was applying to art schools. I applied to two: RISD in the US, and the Accademia di Belli Arti in Florence, which, because it was the oldest art school, I imagined would be good. RISD accepted me, and I never heard back from the Accademia, so off to Providence I went.\\n\\nI\\'d applied for the BFA program at RISD, which meant in effect that I had to go to college again. This was not as strange as it sounds, because I was only 25, and art schools are full of people of different ages. RISD counted me as a transfer sophomore and said I had to do the foundation that summer. The foundation means the classes that everyone has to take in fundamental subjects like drawing, color, and design.\\n\\nToward the end of the summer I got a big surprise: a letter from the Accademia, which had been delayed because they\\'d sent it to Cambridge England instead of Cambridge Massachusetts, inviting me to take the entrance exam in Florence that fall. This was now only weeks away. My nice landlady let me leave my stuff in her attic. I had some money saved from consulting work I\\'d done in grad school; there was probably enough to last a year if I lived cheaply. Now all I had to do was learn Italian.\\n\\nOnly stranieri (foreigners) had to take this entrance exam. In retrospect it may well have been a way of excluding them, because there were so many stranieri attracted by the idea of studying art in Florence that the Italian students would otherwise have been outnumbered. I was in decent shape at painting and drawing from the RISD foundation that summer, but I still don\\'t know how I managed to pass the written exam. I remember that I answered the essay question by writing about Cezanne, and that I cranked up the intellectual level as high as I could to make the most of my limited vocabulary. [2]\\n\\nI\\'m only up to age 25 and already there are such conspicuous patterns. Here I was, yet again about to attend some august institution in the hopes of learning about some prestigious subject, and yet again about to be disappointed. The students and faculty in the painting department at the Accademia were the nicest people you could imagine, but they had long since arrived at an arrangement whereby the students wouldn\\'t require the faculty to teach anything, and in return the faculty wouldn\\'t require the students to learn anything. And at the same time all involved would adhere outwardly to the conventions of a 19th century atelier. We actually had one of those little stoves, fed with kindling, that you see in 19th century studio paintings, and a nude model sitting as close to it as possible without getting burned. Except hardly anyone else painted her besides me. The rest of the students spent their time chatting or occasionally trying to imitate things they\\'d seen in American art magazines.\\n\\nOur model turned out to live just down the street from me. She made a living from a combination of modelling and making fakes for a local antique dealer. She\\'d copy an obscure old painting out of a book, and then he\\'d take the copy and maltreat it to make it look old. [3]\\n\\nWhile I was a student at the Accademia I started painting still lives in my bedroom at night. These paintings were tiny, because the room was, and because I painted them on leftover scraps of canvas, which was all I could afford at the time. Painting still lives is different from painting people, because the subject, as its name suggests, can\\'t move.', mimetype='text/plain', start_char_idx=10841, end_char_idx=15274, metadata_seperator='\\n', text_template='{metadata_str}\\n\\n{content}'), score=0.823051326892001),\n",
       " NodeWithScore(node=TextNode(id_='2793a0b2-457e-4eae-b376-0c79906359ab', embedding=None, metadata={'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='798d3d31-7d21-479b-ba63-0863ee680812', node_type='4', metadata={'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, hash='0c3c3f46cac874b495d944dfc4b920f6b68817dbbb1699ecc955d1fafb2bf87b'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='e5978a51-60fb-49b9-a743-1e9c45cf4f76', node_type='1', metadata={'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, hash='04c4bcbeaa5d8186227f3e4a099a414d42bce393029e4b4c5f478e37cf640ae9'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='3af9fab3-7e31-4242-a227-17dced05e581', node_type='1', metadata={}, hash='725cea4e76028a08701e6dbc435e79e1ccca2047acc8aac89cc408678b451b45')}, metadata_template='{key}: {value}', metadata_separator='\\n', text=\"I kept working on YC till March, to help get that batch of startups through Demo Day, then I checked out pretty completely. (I still talk to alumni and to new startups working on things I'm interested in, but that only takes a few hours a week.)\\n\\nWhat should I do next? Rtm's advice hadn't included anything about that. I wanted to do something completely different, so I decided I'd paint. I wanted to see how good I could get if I really focused on it. So the day after I stopped working on YC, I started painting. I was rusty and it took a while to get back into shape, but it was at least completely engaging. [18]\\n\\nI spent most of the rest of 2014 painting. I'd never been able to work so uninterruptedly before, and I got to be better than I had been. Not good enough, but better. Then in November, right in the middle of a painting, I ran out of steam. Up till that point I'd always been curious to see how the painting I was working on would turn out, but suddenly finishing this one seemed like a chore. So I stopped working on it and cleaned my brushes and haven't painted since. So far anyway.\\n\\nI realize that sounds rather wimpy. But attention is a zero sum game. If you can choose what to work on, and you choose a project that's not the best one (or at least a good one) for you, then it's getting in the way of another project that is. And at 50 there was some opportunity cost to screwing around.\\n\\nI started writing essays again, and wrote a bunch of new ones over the next few months. I even wrote a couple that weren't about startups. Then in March 2015 I started working on Lisp again.\\n\\nThe distinctive thing about Lisp is that its core is a language defined by writing an interpreter in itself. It wasn't originally intended as a programming language in the ordinary sense. It was meant to be a formal model of computation, an alternative to the Turing machine. If you want to write an interpreter for a language in itself, what's the minimum set of predefined operators you need? The Lisp that John McCarthy invented, or more accurately discovered, is an answer to that question. [19]\\n\\nMcCarthy didn't realize this Lisp could even be used to program computers till his grad student Steve Russell suggested it. Russell translated McCarthy's interpreter into IBM 704 machine language, and from that point Lisp started also to be a programming language in the ordinary sense. But its origins as a model of computation gave it a power and elegance that other languages couldn't match. It was this that attracted me in college, though I didn't understand why at the time.\\n\\nMcCarthy's 1960 Lisp did nothing more than interpret Lisp expressions. It was missing a lot of things you'd want in a programming language. So these had to be added, and when they were, they weren't defined using McCarthy's original axiomatic approach. That wouldn't have been feasible at the time. McCarthy tested his interpreter by hand-simulating the execution of programs. But it was already getting close to the limit of interpreters you could test that way — indeed, there was a bug in it that McCarthy had overlooked. To test a more complicated interpreter, you'd have had to run it, and computers then weren't powerful enough.\\n\\nNow they are, though. Now you could continue using McCarthy's axiomatic approach till you'd defined a complete programming language. And as long as every change you made to McCarthy's Lisp was a discoveredness-preserving transformation, you could, in principle, end up with a complete language that had this quality. Harder to do than to talk about, of course, but if it was possible in principle, why not try? So I decided to take a shot at it. It took 4 years, from March 26, 2015 to October 12, 2019. It was fortunate that I had a precisely defined goal, or it would have been hard to keep at it for so long.\\n\\nI wrote this new Lisp, called Bel, in itself in Arc. That may sound like a contradiction, but it's an indication of the sort of trickery I had to engage in to make this work. By means of an egregious collection of hacks I managed to make something close enough to an interpreter written in itself that could actually run. Not fast, but fast enough to test.\\n\\nI had to ban myself from writing essays during most of this time, or I'd never have finished. In late 2015 I spent 3 months writing essays, and when I went back to working on Bel I could barely understand the code.\", mimetype='text/plain', start_char_idx=60783, end_char_idx=65193, metadata_seperator='\\n', text_template='{metadata_str}\\n\\n{content}'), score=0.8224947294630384),\n",
       " NodeWithScore(node=TextNode(id_='e5978a51-60fb-49b9-a743-1e9c45cf4f76', embedding=None, metadata={'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='798d3d31-7d21-479b-ba63-0863ee680812', node_type='4', metadata={'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, hash='0c3c3f46cac874b495d944dfc4b920f6b68817dbbb1699ecc955d1fafb2bf87b'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='bb017ef8-562b-49bd-95d3-ca13452fb4e9', node_type='1', metadata={'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, hash='c3754e3071672ecf4820d91b195fdda233136ea9d5ba19a80d76448662f80eef'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='2793a0b2-457e-4eae-b376-0c79906359ab', node_type='1', metadata={}, hash='25f1cdf846d1204cdae80bd7a6a63681cf58cc7145d0a0a929a25eedd1d6d71c')}, metadata_template='{key}: {value}', metadata_separator='\\n', text='It was very engaging work, because their problems were quite varied, and the good founders were very effective. If you were trying to learn the most you could about startups in the shortest possible time, you couldn\\'t have picked a better way to do it.\\n\\nThere were parts of the job I didn\\'t like. Disputes between cofounders, figuring out when people were lying to us, fighting with people who maltreated the startups, and so on. But I worked hard even at the parts I didn\\'t like. I was haunted by something Kevin Hale once said about companies: \"No one works harder than the boss.\" He meant it both descriptively and prescriptively, and it was the second part that scared me. I wanted YC to be good, so if how hard I worked set the upper bound on how hard everyone else worked, I\\'d better work very hard.\\n\\nOne day in 2010, when he was visiting California for interviews, Robert Morris did something astonishing: he offered me unsolicited advice. I can only remember him doing that once before. One day at Viaweb, when I was bent over double from a kidney stone, he suggested that it would be a good idea for him to take me to the hospital. That was what it took for Rtm to offer unsolicited advice. So I remember his exact words very clearly. \"You know,\" he said, \"you should make sure Y Combinator isn\\'t the last cool thing you do.\"\\n\\nAt the time I didn\\'t understand what he meant, but gradually it dawned on me that he was saying I should quit. This seemed strange advice, because YC was doing great. But if there was one thing rarer than Rtm offering advice, it was Rtm being wrong. So this set me thinking. It was true that on my current trajectory, YC would be the last thing I did, because it was only taking up more of my attention. It had already eaten Arc, and was in the process of eating essays too. Either YC was my life\\'s work or I\\'d have to leave eventually. And it wasn\\'t, so I would.\\n\\nIn the summer of 2012 my mother had a stroke, and the cause turned out to be a blood clot caused by colon cancer. The stroke destroyed her balance, and she was put in a nursing home, but she really wanted to get out of it and back to her house, and my sister and I were determined to help her do it. I used to fly up to Oregon to visit her regularly, and I had a lot of time to think on those flights. On one of them I realized I was ready to hand YC over to someone else.\\n\\nI asked Jessica if she wanted to be president, but she didn\\'t, so we decided we\\'d try to recruit Sam Altman. We talked to Robert and Trevor and we agreed to make it a complete changing of the guard. Up till that point YC had been controlled by the original LLC we four had started. But we wanted YC to last for a long time, and to do that it couldn\\'t be controlled by the founders. So if Sam said yes, we\\'d let him reorganize YC. Robert and I would retire, and Jessica and Trevor would become ordinary partners.\\n\\nWhen we asked Sam if he wanted to be president of YC, initially he said no. He wanted to start a startup to make nuclear reactors. But I kept at it, and in October 2013 he finally agreed. We decided he\\'d take over starting with the winter 2014 batch. For the rest of 2013 I left running YC more and more to Sam, partly so he could learn the job, and partly because I was focused on my mother, whose cancer had returned.\\n\\nShe died on January 15, 2014. We knew this was coming, but it was still hard when it did.\\n\\nI kept working on YC till March, to help get that batch of startups through Demo Day, then I checked out pretty completely. (I still talk to alumni and to new startups working on things I\\'m interested in, but that only takes a few hours a week.)\\n\\nWhat should I do next? Rtm\\'s advice hadn\\'t included anything about that. I wanted to do something completely different, so I decided I\\'d paint. I wanted to see how good I could get if I really focused on it. So the day after I stopped working on YC, I started painting. I was rusty and it took a while to get back into shape, but it was at least completely engaging. [18]\\n\\nI spent most of the rest of 2014 painting. I\\'d never been able to work so uninterruptedly before, and I got to be better than I had been.', mimetype='text/plain', start_char_idx=57383, end_char_idx=61540, metadata_seperator='\\n', text_template='{metadata_str}\\n\\n{content}'), score=0.8197649990461686),\n",
       " NodeWithScore(node=TextNode(id_='6d9b4641-53ae-4a6d-989a-20cadd9415e5', embedding=None, metadata={'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='798d3d31-7d21-479b-ba63-0863ee680812', node_type='4', metadata={'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, hash='0c3c3f46cac874b495d944dfc4b920f6b68817dbbb1699ecc955d1fafb2bf87b'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='9397dfe3-9a96-4036-a01a-1225ca2b8310', node_type='1', metadata={}, hash='692d74391b6ad961556d19669374a543e4752bab10e415d8eeeda9ccf682a597')}, metadata_template='{key}: {value}', metadata_separator='\\n', text='What I Worked On\\n\\nFebruary 2021\\n\\nBefore college the two main things I worked on, outside of school, were writing and programming. I didn\\'t write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.\\n\\nThe first programs I tried writing were on the IBM 1401 that our school district used for what was then called \"data processing.\" This was in 9th grade, so I was 13 or 14. The school district\\'s 1401 happened to be in the basement of our junior high school, and my friend Rich Draves and I got permission to use it. It was like a mini Bond villain\\'s lair down there, with all these alien-looking machines — CPU, disk drives, printer, card reader — sitting up on a raised floor under bright fluorescent lights.\\n\\nThe language we used was an early version of Fortran. You had to type programs on punch cards, then stack them in the card reader and press a button to load the program into memory and run it. The result would ordinarily be to print something on the spectacularly loud printer.\\n\\nI was puzzled by the 1401. I couldn\\'t figure out what to do with it. And in retrospect there\\'s not much I could have done with it. The only form of input to programs was data stored on punched cards, and I didn\\'t have any data stored on punched cards. The only other option was to do things that didn\\'t rely on any input, like calculate approximations of pi, but I didn\\'t know enough math to do anything interesting of that type. So I\\'m not surprised I can\\'t remember any programs I wrote, because they can\\'t have done much. My clearest memory is of the moment I learned it was possible for programs not to terminate, when one of mine didn\\'t. On a machine without time-sharing, this was a social as well as a technical error, as the data center manager\\'s expression made clear.\\n\\nWith microcomputers, everything changed. Now you could have a computer sitting right in front of you, on a desk, that could respond to your keystrokes as it was running instead of just churning through a stack of punch cards and then stopping. [1]\\n\\nThe first of my friends to get a microcomputer built it himself. It was sold as a kit by Heathkit. I remember vividly how impressed and envious I felt watching him sitting in front of it, typing programs right into the computer.\\n\\nComputers were expensive in those days and it took me years of nagging before I convinced my father to buy one, a TRS-80, in about 1980. The gold standard then was the Apple II, but a TRS-80 was good enough. This was when I really started programming. I wrote simple games, a program to predict how high my model rockets would fly, and a word processor that my father used to write at least one book. There was only room in memory for about 2 pages of text, so he\\'d write 2 pages at a time and then print them out, but it was a lot better than a typewriter.\\n\\nThough I liked programming, I didn\\'t plan to study it in college. In college I was going to study philosophy, which sounded much more powerful. It seemed, to my naive high school self, to be the study of the ultimate truths, compared to which the things studied in other fields would be mere domain knowledge. What I discovered when I got to college was that the other fields took up so much of the space of ideas that there wasn\\'t much left for these supposed ultimate truths. All that seemed left for philosophy were edge cases that people in other fields felt could safely be ignored.\\n\\nI couldn\\'t have put this into words when I was 18. All I knew at the time was that I kept taking philosophy courses and they kept being boring. So I decided to switch to AI.\\n\\nAI was in the air in the mid 1980s, but there were two things especially that made me want to work on it: a novel by Heinlein called The Moon is a Harsh Mistress, which featured an intelligent computer called Mike, and a PBS documentary that showed Terry Winograd using SHRDLU. I haven\\'t tried rereading The Moon is a Harsh Mistress, so I don\\'t know how well it has aged, but when I read it I was drawn entirely into its world. It seemed only a matter of time before we\\'d have Mike, and when I saw Winograd using SHRDLU, it seemed like that time would be a few years at most. All you had to do was teach SHRDLU more words.', mimetype='text/plain', start_char_idx=0, end_char_idx=4365, metadata_seperator='\\n', text_template='{metadata_str}\\n\\n{content}'), score=0.817735099719819),\n",
       " NodeWithScore(node=TextNode(id_='3bac5d02-c6a0-4e92-9e23-9286254fa682', embedding=None, metadata={'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='798d3d31-7d21-479b-ba63-0863ee680812', node_type='4', metadata={'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, hash='0c3c3f46cac874b495d944dfc4b920f6b68817dbbb1699ecc955d1fafb2bf87b'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='9cde9b0c-c984-4a04-942a-787c209e1464', node_type='1', metadata={'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, hash='45b44307b74ff35b46be9755b5e062f59d89e59d1c3034dbd958e02a0e80599e'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='74c8db56-88a2-40fd-945d-60ec37b14cf0', node_type='1', metadata={}, hash='520fad04e6c34339505833f62936793a9d722dce0202a5f034329ba60c92b430')}, metadata_template='{key}: {value}', metadata_separator='\\n', text=\"The next year, from the summer of 1998 to the summer of 1999, must have been the least productive of my life. I didn't realize it at the time, but I was worn out from the effort and stress of running Viaweb. For a while after I got to California I tried to continue my usual m.o. of programming till 3 in the morning, but fatigue combined with Yahoo's prematurely aged culture and grim cube farm in Santa Clara gradually dragged me down. After a few months it felt disconcertingly like working at Interleaf.\\n\\nYahoo had given us a lot of options when they bought us. At the time I thought Yahoo was so overvalued that they'd never be worth anything, but to my astonishment the stock went up 5x in the next year. I hung on till the first chunk of options vested, then in the summer of 1999 I left. It had been so long since I'd painted anything that I'd half forgotten why I was doing this. My brain had been entirely full of software and men's shirts for 4 years. But I had done this to get rich so I could paint, I reminded myself, and now I was rich, so I should go paint.\\n\\nWhen I said I was leaving, my boss at Yahoo had a long conversation with me about my plans. I told him all about the kinds of pictures I wanted to paint. At the time I was touched that he took such an interest in me. Now I realize it was because he thought I was lying. My options at that point were worth about $2 million a month. If I was leaving that kind of money on the table, it could only be to go and start some new startup, and if I did, I might take people with me. This was the height of the Internet Bubble, and Yahoo was ground zero of it. My boss was at that moment a billionaire. Leaving then to start a new startup must have seemed to him an insanely, and yet also plausibly, ambitious plan.\\n\\nBut I really was quitting to paint, and I started immediately. There was no time to lose. I'd already burned 4 years getting rich. Now when I talk to founders who are leaving after selling their companies, my advice is always the same: take a vacation. That's what I should have done, just gone off somewhere and done nothing for a month or two, but the idea never occurred to me.\\n\\nSo I tried to paint, but I just didn't seem to have any energy or ambition. Part of the problem was that I didn't know many people in California. I'd compounded this problem by buying a house up in the Santa Cruz Mountains, with a beautiful view but miles from anywhere. I stuck it out for a few more months, then in desperation I went back to New York, where unless you understand about rent control you'll be surprised to hear I still had my apartment, sealed up like a tomb of my old life. Idelle was in New York at least, and there were other people trying to paint there, even though I didn't know any of them.\\n\\nWhen I got back to New York I resumed my old life, except now I was rich. It was as weird as it sounds. I resumed all my old patterns, except now there were doors where there hadn't been. Now when I was tired of walking, all I had to do was raise my hand, and (unless it was raining) a taxi would stop to pick me up. Now when I walked past charming little restaurants I could go in and order lunch. It was exciting for a while. Painting started to go better. I experimented with a new kind of still life where I'd paint one painting in the old way, then photograph it and print it, blown up, on canvas, and then use that as the underpainting for a second still life, painted from the same objects (which hopefully hadn't rotted yet).\\n\\nMeanwhile I looked for an apartment to buy. Now I could actually choose what neighborhood to live in. Where, I asked myself and various real estate agents, is the Cambridge of New York? Aided by occasional visits to actual Cambridge, I gradually realized there wasn't one. Huh.\\n\\nAround this time, in the spring of 2000, I had an idea. It was clear from our experience with Viaweb that web apps were the future. Why not build a web app for making web apps? Why not let people edit code on our server through the browser, and then host the resulting applications for them? [9] You could run all sorts of services on the servers that these applications could use just by making an API call: making and receiving phone calls, manipulating images, taking credit card payments, etc.\", mimetype='text/plain', start_char_idx=35841, end_char_idx=40135, metadata_seperator='\\n', text_template='{metadata_str}\\n\\n{content}'), score=0.8129469543750788),\n",
       " NodeWithScore(node=TextNode(id_='7748cd46-2588-4ec1-8220-2a8c85e684c7', embedding=None, metadata={'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='798d3d31-7d21-479b-ba63-0863ee680812', node_type='4', metadata={'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, hash='0c3c3f46cac874b495d944dfc4b920f6b68817dbbb1699ecc955d1fafb2bf87b'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='3af9fab3-7e31-4242-a227-17dced05e581', node_type='1', metadata={'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, hash='41840768ce50c6b638bbfa32151ae2289b82f44fe0c0fb46151ead709f5fb26f'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='30aa4c51-42e6-4526-a7a6-000a5d129940', node_type='1', metadata={}, hash='9aaeaab1077c1a7faf97ded69d2d037e18ed9d3cb0e639c24a0282b284a8a742')}, metadata_template='{key}: {value}', metadata_separator='\\n', text=\"[3] I lived at Piazza San Felice 4, so my walk to the Accademia went straight down the spine of old Florence: past the Pitti, across the bridge, past Orsanmichele, between the Duomo and the Baptistery, and then up Via Ricasoli to Piazza San Marco. I saw Florence at street level in every possible condition, from empty dark winter evenings to sweltering summer days when the streets were packed with tourists.\\n\\n[4] You can of course paint people like still lives if you want to, and they're willing. That sort of portrait is arguably the apex of still life painting, though the long sitting does tend to produce pained expressions in the sitters.\\n\\n[5] Interleaf was one of many companies that had smart people and built impressive technology, and yet got crushed by Moore's Law. In the 1990s the exponential growth in the power of commodity (i.e. Intel) processors rolled up high-end, special-purpose hardware and software companies like a bulldozer.\\n\\n[6] The signature style seekers at RISD weren't specifically mercenary. In the art world, money and coolness are tightly coupled. Anything expensive comes to be seen as cool, and anything seen as cool will soon become equally expensive.\\n\\n[7] Technically the apartment wasn't rent-controlled but rent-stabilized, but this is a refinement only New Yorkers would know or care about. The point is that it was really cheap, less than half market price.\\n\\n[8] Most software you can launch as soon as it's done. But when the software is an online store builder and you're hosting the stores, if you don't have any users yet, that fact will be painfully obvious. So before we could launch publicly we had to launch privately, in the sense of recruiting an initial set of users and making sure they had decent-looking stores.\\n\\n[9] We'd had a code editor in Viaweb for users to define their own page styles. They didn't know it, but they were editing Lisp expressions underneath. But this wasn't an app editor, because the code ran when the merchants' sites were generated, not when shoppers visited them.\\n\\n[10] This was the first instance of what is now a familiar experience, and so was what happened next, when I read the comments and found they were full of angry people. How could I claim that Lisp was better than other languages? Weren't they all Turing complete? People who see the responses to essays I write sometimes tell me how sorry they feel for me, but I'm not exaggerating when I reply that it has always been like this, since the very beginning. It comes with the territory. An essay must tell readers things they don't already know, and some people dislike being told such things.\\n\\n[11] People put plenty of stuff on the internet in the 90s of course, but putting something online is not the same as publishing it online. Publishing online means you treat the online version as the (or at least a) primary version.\\n\\n[12] There is a general lesson here that our experience with Y Combinator also teaches: Customs continue to constrain you long after the restrictions that caused them have disappeared. Customary VC practice had once, like the customs about publishing essays, been based on real constraints. Startups had once been much more expensive to start, and proportionally rare. Now they could be cheap and common, but the VCs' customs still reflected the old world, just as customs about writing essays still reflected the constraints of the print era.\\n\\nWhich in turn implies that people who are independent-minded (i.e. less influenced by custom) will have an advantage in fields affected by rapid change (where customs are more likely to be obsolete).\\n\\nHere's an interesting point, though: you can't always predict which fields will be affected by rapid change. Obviously software and venture capital will be, but who would have predicted that essay writing would be?\\n\\n[13] Y Combinator was not the original name. At first we were called Cambridge Seed. But we didn't want a regional name, in case someone copied us in Silicon Valley, so we renamed ourselves after one of the coolest tricks in the lambda calculus, the Y combinator.\\n\\nI picked orange as our color partly because it's the warmest, and partly because no VC used it. In 2005 all the VCs used staid colors like maroon, navy blue, and forest green, because they were trying to appeal to LPs, not founders. The YC logo itself is an inside joke: the Viaweb logo had been a white V on a red circle, so I made the YC logo a white Y on an orange square.\", mimetype='text/plain', start_char_idx=68036, end_char_idx=72513, metadata_seperator='\\n', text_template='{metadata_str}\\n\\n{content}'), score=0.8103934429245039),\n",
       " NodeWithScore(node=TextNode(id_='dd56b836-1549-416e-824b-88989aca584c', embedding=None, metadata={'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='798d3d31-7d21-479b-ba63-0863ee680812', node_type='4', metadata={'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, hash='0c3c3f46cac874b495d944dfc4b920f6b68817dbbb1699ecc955d1fafb2bf87b'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='9397dfe3-9a96-4036-a01a-1225ca2b8310', node_type='1', metadata={'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, hash='a54176e403896d518137de1f4ac5cb2b42c68e364af03e7eaa4b102a6a166524'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='f01289fa-87f1-4dcd-b0cc-6f44ae51629c', node_type='1', metadata={}, hash='14da015be7821a729e7a5d22fcca89904fcaa4f8cb586c20e8cf4baf5b318534')}, metadata_template='{key}: {value}', metadata_separator='\\n', text='So I looked around to see what I could salvage from the wreckage of my plans, and there was Lisp. I knew from experience that Lisp was interesting for its own sake and not just for its association with AI, even though that was the main reason people cared about it at the time. So I decided to focus on Lisp. In fact, I decided to write a book about Lisp hacking. It\\'s scary to think how little I knew about Lisp hacking when I started writing that book. But there\\'s nothing like writing a book about something to help you learn it. The book, On Lisp, wasn\\'t published till 1993, but I wrote much of it in grad school.\\n\\nComputer Science is an uneasy alliance between two halves, theory and systems. The theory people prove things, and the systems people build things. I wanted to build things. I had plenty of respect for theory — indeed, a sneaking suspicion that it was the more admirable of the two halves — but building things seemed so much more exciting.\\n\\nThe problem with systems work, though, was that it didn\\'t last. Any program you wrote today, no matter how good, would be obsolete in a couple decades at best. People might mention your software in footnotes, but no one would actually use it. And indeed, it would seem very feeble work. Only people with a sense of the history of the field would even realize that, in its time, it had been good.\\n\\nThere were some surplus Xerox Dandelions floating around the computer lab at one point. Anyone who wanted one to play around with could have one. I was briefly tempted, but they were so slow by present standards; what was the point? No one else wanted one either, so off they went. That was what happened to systems work.\\n\\nI wanted not just to build things, but to build things that would last.\\n\\nIn this dissatisfied state I went in 1988 to visit Rich Draves at CMU, where he was in grad school. One day I went to visit the Carnegie Institute, where I\\'d spent a lot of time as a kid. While looking at a painting there I realized something that might seem obvious, but was a big surprise to me. There, right on the wall, was something you could make that would last. Paintings didn\\'t become obsolete. Some of the best ones were hundreds of years old.\\n\\nAnd moreover this was something you could make a living doing. Not as easily as you could by writing software, of course, but I thought if you were really industrious and lived really cheaply, it had to be possible to make enough to survive. And as an artist you could be truly independent. You wouldn\\'t have a boss, or even need to get research funding.\\n\\nI had always liked looking at paintings. Could I make them? I had no idea. I\\'d never imagined it was even possible. I knew intellectually that people made art — that it didn\\'t just appear spontaneously — but it was as if the people who made it were a different species. They either lived long ago or were mysterious geniuses doing strange things in profiles in Life magazine. The idea of actually being able to make art, to put that verb before that noun, seemed almost miraculous.\\n\\nThat fall I started taking art classes at Harvard. Grad students could take classes in any department, and my advisor, Tom Cheatham, was very easy going. If he even knew about the strange classes I was taking, he never said anything.\\n\\nSo now I was in a PhD program in computer science, yet planning to be an artist, yet also genuinely in love with Lisp hacking and working away at On Lisp. In other words, like many a grad student, I was working energetically on multiple projects that were not my thesis.\\n\\nI didn\\'t see a way out of this situation. I didn\\'t want to drop out of grad school, but how else was I going to get out? I remember when my friend Robert Morris got kicked out of Cornell for writing the internet worm of 1988, I was envious that he\\'d found such a spectacular way to get out of grad school.\\n\\nThen one day in April 1990 a crack appeared in the wall. I ran into professor Cheatham and he asked if I was far enough along to graduate that June. I didn\\'t have a word of my dissertation written, but in what must have been the quickest bit of thinking in my life, I decided to take a shot at writing one in the 5 weeks or so that remained before the deadline, reusing parts of On Lisp where I could, and I was able to respond, with no perceptible delay \"Yes, I think so. I\\'ll give you something to read in a few days.\"\\n\\nI picked applications of continuations as the topic.', mimetype='text/plain', start_char_idx=7164, end_char_idx=11601, metadata_seperator='\\n', text_template='{metadata_str}\\n\\n{content}'), score=0.8098668934256743),\n",
       " NodeWithScore(node=TextNode(id_='86d9d9e4-f480-43a0-9eb4-a898578aabe3', embedding=None, metadata={'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='798d3d31-7d21-479b-ba63-0863ee680812', node_type='4', metadata={'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, hash='0c3c3f46cac874b495d944dfc4b920f6b68817dbbb1699ecc955d1fafb2bf87b'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='74c8db56-88a2-40fd-945d-60ec37b14cf0', node_type='1', metadata={'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, hash='662d7594b4c311f5c96528558e24976429a1f10e4493fbdea717680049136922'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='63e74b14-d063-43b1-9c16-6f5646a9b81f', node_type='1', metadata={}, hash='bad175b4bfd702efe27ed34d642be18b0cce43fab7a409cd71bb89c8e96c9467')}, metadata_template='{key}: {value}', metadata_separator='\\n', text=\"I was invited to give a talk at a Lisp conference, so I gave one about how we'd used Lisp at Viaweb. Afterward I put a postscript file of this talk online, on paulgraham.com, which I'd created years before using Viaweb but had never used for anything. In one day it got 30,000 page views. What on earth had happened? The referring urls showed that someone had posted it on Slashdot. [10]\\n\\nWow, I thought, there's an audience. If I write something and put it on the web, anyone can read it. That may seem obvious now, but it was surprising then. In the print era there was a narrow channel to readers, guarded by fierce monsters known as editors. The only way to get an audience for anything you wrote was to get it published as a book, or in a newspaper or magazine. Now anyone could publish anything.\\n\\nThis had been possible in principle since 1993, but not many people had realized it yet. I had been intimately involved with building the infrastructure of the web for most of that time, and a writer as well, and it had taken me 8 years to realize it. Even then it took me several years to understand the implications. It meant there would be a whole new generation of essays. [11]\\n\\nIn the print era, the channel for publishing essays had been vanishingly small. Except for a few officially anointed thinkers who went to the right parties in New York, the only people allowed to publish essays were specialists writing about their specialties. There were so many essays that had never been written, because there had been no way to publish them. Now they could be, and I was going to write them. [12]\\n\\nI've worked on several different things, but to the extent there was a turning point where I figured out what to work on, it was when I started publishing essays online. From then on I knew that whatever else I did, I'd always write essays too.\\n\\nI knew that online essays would be a marginal medium at first. Socially they'd seem more like rants posted by nutjobs on their GeoCities sites than the genteel and beautifully typeset compositions published in The New Yorker. But by this point I knew enough to find that encouraging instead of discouraging.\\n\\nOne of the most conspicuous patterns I've noticed in my life is how well it has worked, for me at least, to work on things that weren't prestigious. Still life has always been the least prestigious form of painting. Viaweb and Y Combinator both seemed lame when we started them. I still get the glassy eye from strangers when they ask what I'm writing, and I explain that it's an essay I'm going to publish on my web site. Even Lisp, though prestigious intellectually in something like the way Latin is, also seems about as hip.\\n\\nIt's not that unprestigious types of work are good per se. But when you find yourself drawn to some kind of work despite its current lack of prestige, it's a sign both that there's something real to be discovered there, and that you have the right kind of motives. Impure motives are a big danger for the ambitious. If anything is going to lead you astray, it will be the desire to impress people. So while working on things that aren't prestigious doesn't guarantee you're on the right track, it at least guarantees you're not on the most common type of wrong one.\\n\\nOver the next several years I wrote lots of essays about all kinds of different topics. O'Reilly reprinted a collection of them as a book, called Hackers & Painters after one of the essays in it. I also worked on spam filters, and did some more painting. I used to have dinners for a group of friends every thursday night, which taught me how to cook for groups. And I bought another building in Cambridge, a former candy factory (and later, twas said, porn studio), to use as an office.\\n\\nOne night in October 2003 there was a big party at my house. It was a clever idea of my friend Maria Daniels, who was one of the thursday diners. Three separate hosts would all invite their friends to one party. So for every guest, two thirds of the other guests would be people they didn't know but would probably like. One of the guests was someone I didn't know but would turn out to like a lot: a woman called Jessica Livingston. A couple days later I asked her out.\\n\\nJessica was in charge of marketing at a Boston investment bank. This bank thought it understood startups, but over the next year, as she met friends of mine from the startup world, she was surprised how different reality was.\", mimetype='text/plain', start_char_idx=42978, end_char_idx=47421, metadata_seperator='\\n', text_template='{metadata_str}\\n\\n{content}'), score=0.8069900228038202)]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nodes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prompt compression using llmlingua2\n",
    "LLMLingua2's claim to fame is its ability to achieve performant compression using a small prompt compression method trained via data distillation from GPT-4 for token classification! This performant compression comes with a performance bump of 3x-6x\n",
    "> https://aclanthology.org/2024.findings-acl.57/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "compressor_llmlingua2 = LongLLMLinguaPostprocessor(\n",
    "    model_name=\"microsoft/llmlingua-2-xlm-roberta-large-meetingbank\",\n",
    "    device_map=\"mps\",  # Mac users rejoice!\n",
    "    use_llmlingua2=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Token indices sequence length is longer than the specified maximum sequence length for this model (8451 > 512). Running this sequence through the model will result in indexing errors\n"
     ]
    }
   ],
   "source": [
    "from llama_index.core.schema import QueryBundle\n",
    "\n",
    "results = compressor_llmlingua2._postprocess_nodes(\n",
    "    nodes, query_bundle=QueryBundle(query_str=query)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "remember friend Robert Morris kicked out of Cornell for writing internet worm 1988 envious found spectacular way to get out grad school April 1990 crack appeared in wall ran into professor Cheatham asked if far enough along to graduate June didn't have word dissertation written decided to writing one in 5 weeks before deadline reusing parts On Lisp no delay give something to read in few days.\" picked applications of continuations topic should have written about macros embedded languages world barely explored wanted to get out of grad school rapidly written dissertation sufficed barely applying to art schools applied to two RISD Accademia di Belli Arti in Florence oldest art school RISD accepted never heard back from Accademia off to Providence applied for BFA program at RISD had to go to college again 25 art schools full of different ages RISD counted transfer sophomore to do foundation summer foundation classes fundamental subjects like drawing color design end of summer got surprise letter from Accademia delayed Cambridge England inviting to take entrance exam in Florence fall weeks away landlady let leave stuff in attichad money saved from consulting work in grad school enough to last a year if lived cheaply. had to learn Italian. Only stranieri (foreigners) take entrance exam. may have excluding them many stranieri attracted studying art in Florence Italian students outnumbered. decent shape at painting and drawing from RISD foundation summer don't pass written exam. answered essay question by writing about Cezanne cranked up intellectual level high most of limited vocabulary. only up to age 25 conspicuous patterns. about to attend august institution learning prestigious subject disappointed. students and faculty in painting department at Accademia nicest arrangement students wouldn't require faculty to teach faculty't require students to learn anything all adhere to conventions of 19th century atelier. had stoves fed with kindling 19th century studio paintings nude model sitting close to without hardly anyone painted her besides me students spent chatting or imitate American art magazines. model down street from me made living from modelling making fakes for local antique dealer copy obscure old painting look old. student at Accademia started painting still lives in my bedroom at nightpaintings tiny room was painted on leftover scraps canvas all could afford Painting still lives different from painting people subject suggests can't move.I kept working on YC till March help get startups through Demo Day then checked out completely still talk to alumni new startups takes few hours a week What do next? Rtm's advice't included wanted something different decided'd paint wanted to see how good get if focused day after stopped working on YC started painting was rusty took while to get back into shape but engaging. spent most rest of 2014 painting never uninterruptedly got better Not good enough but better in November middle of painting ran out of steam curious to painting finishing seemed like a chore stopped working cleaned brushes haven't painted since attention is zero sum game If choose what to work on project not best (or good in way of another project at 50 opportunity cost to screwing around started writing essays again wrote new ones months wrote couple't about startups in March 2015 started working on Lisp again distinctive Lisp core is language defined by writing interpreter't originally intended as programming languagemeant formal model of computation alternative to Turing machine write interpreter for language minimum predefined operators need? Lisp John McCarthy invented discovered answer McCarthy didn't realize Lisp used program computers till grad student Steve Russell suggested Russell translated McCarthy interpreter into IBM 704 machine language Lisp started programming language origins model computation gave power elegance other languages couldn't match attracted me in college didn't understand McCarthy's 1960 Lisp interpret Lisp expressions missing things programming language had to be added weren't defined using McCarthy's original axiomatic approach feasible McCarthy tested interpreter by hand-simulating execution programs close to limit of interpreters bug overlooked test more complicated interpreter run computers weren't powerful enough Now are continue using McCarthy's axiomatic approach till defined complete programming language every change to McCarthy's Lisp discoveredness-preserving transformation could end complete language quality Harder to do talk if possible principle why not try? decided to shot took 4 years March 26, 2015 to October 12, 2019. fortunate had precisely defined goal or hard to keepwrote new Lisp Bel, in Arc. contradiction indication of trickery to make work. By egregious collection of hacks managed to make something close to interpreter in could run Not fast but fast enough to test. had to ban myself from writing essays time or'd never finished. late 2015 spent 3 months writing essays back to working on Bel barely understand code.It engaging work problems varied good founders effective trying to learn most about startups in shortest time better way. parts job didn't like Disputes between cofounders figuring people lying fighting with people maltreated startups worked hard even haunted by Kevin Hale \"No one works harder than boss.\" descriptively second part scared me. wanted YC to be good if hard I worked set upper bound on work hard. in 2010, visiting California for interviews Robert Morris offered unsolicited advice remember once before Viaweb bent over from kidney stone suggested good idea for to take me to hospital unsolicited advice remember words Y Combinator isn't last cool thing you do.\" didn't understand dawned he was saying I should quit.strange advice YC doing great. thing rarer than Rtm offering advice Rtm being wrong set me thinking true current trajectory YC last thing I did taking up more attention eaten Arc eating essays Either YC was life's work or I leave wasn't would summer 2012 mother had stroke cause blood clot colon cancer stroke destroyed her balance put in nursing home wanted to get back to house sister and I determined to help fly to Oregon visit her time to think flights ready to hand YC over asked Jessica president didn't decided recruit Sam Altman talked to Robert and Trevor agreed complete changing of guard YC controlled by original LLC wanted YC to last long time couldn't be controlled by founders if Sam said yes let him reorganize YC Robert and I retire Jessica and Trevor become ordinary partners asked Sam president YC initially said no wanted to start startup make nuclear reactors kept at October 2013 agreed decided he'd take over starting winter 2014 batch rest of 2013 left running YC to Sam he learn job focused on mother cancer returned died January 15, 2014. knew was coming hardkept working on YC till March get startups through Demo Day then checked out completely still talk to alumni new startups takes few hours a week What next? Rtm's advice included wanted something different decided paint see good focused day after stopped working on YC started painting was rusty took to get shape but engaging. spent rest of 2014 painting never uninterruptedly got better Worked On February 2021 Before college main worked on writing and programming didn't write essays wrote short stories stories awful plot characters with strong feelings first programs tried writing on IBM 1401 school district used for \"data processing.\" in 9th grade 13 or 14. school district's 1401 in basement of junior high school friend Rich Draves and I permission to use like mini Bond villain's lair alien-looking machines CPU disk drives printer card reader on raised floor under fluorescent lights language used was early version Fortran type programs on punch cards stack in card reader to load program into memory run result print on loud printer puzzled by 1401 couldn't figure out what to do with it not much could have done with itonly input to programs was data on punched cards I didn't have data on other option was to do things input like calculate approximations of pi didn't know enough math not surprised can't remember programs I wrote can't have done much clearest memory of moment learned possible for programs not to terminate one mine didn't On machine without time-sharing was social technical error data center manager. microcomputers everything changed computer respond to keystrokes running instead of punch cards stopping first to get microcomputer built it himself sold as kit by Heathkit remember impressed envious watching him typing programs into Computers were expensive took years of nagging before convinced father to buy one TRS-80, in 1980 gold standard was Apple II TRS-80 was good enough started programming wrote simple games, program to predict model rockets fly word processor father used to write book only room in memory for 2 pages of text write 2 pages at print better than typewriter liked programming didn't plan to study it in college In to study philosophy more powerful study of ultimate truths domain knowledgediscovered college other fields took up space ideas much left for ultimate truths left for philosophy were edge cases ignored.'t words when 18. kept taking philosophy courses boring decided to switch to AI. AI in air in mid 1980s two things made want to work: novel by Heinlein The Moon is a Harsh Mistress featured intelligent computer Mike PBS documentary showed Terry Winograd using SHRDLU. rereading The Moon is a Harsh Mistress read drawn into its world seemed matter of time before Mike saw Winograd using SHRDLU time few years teach SHRDLU more words.The next year from summer 1998 to summer 1999 least productive worn out from effort stress running Viaweb. after California tried to continue programming till 3 morning fatigue Yahoo's prematurely aged culture grim cube farm in Santa Clara dragged me down After few months felt like working at Interleaf. Yahoo given options bought us thought Yahoo overvalued stock went up 5x next year hung on till first chunk of options vested then summer of 1999 left. long since painted anything half forgotten why brain full of software and men's shirts for 4 years.done this to get rich so could paint reminded now was rich should go paint. said leaving boss at Yahoo had long conversation about plans told him about pictures wanted to paint touched he took interest in Now realize because he thought I was lying. options worth about $2 million a month leaving money on table could be to start new startup might take people height of Internet Bubble Yahoo was ground zero boss was a billionaire Leaving to start new startup seemed insanely ambitious plan. quitting to paint started immediately no time to lose burned 4 years getting rich to founders leaving after selling companies advice: take a vacation should have done for month idea never occurred tried to paint didn't energy or ambition problem didn't know many people in California compounded problem by buying house in Santa Cruz Mountains beautiful view miles from anywhere stuck it out for few more months desperation went back to New York still had apartment old life Idelle in New York other people trying to paint didn't know back to New York resumed old life except now was rich resumed old patterns except now doors where there hadn't been.tired of walking raise hand raining taxi stop pick me up walked past restaurants order lunch exciting for Painting better experimented with new still life paint one painting old way photograph print on canvas use as underpainting for second still life from same objects rotted looked for apartment choose neighborhood asked real estate agents Cambridge of New York? occasional visits to Cambridge realized wasn't one spring of 2000, had idea Viaweb web apps were future build web app for making web apps let people edit code on server through browser host resulting applications for? run services on servers applications by making API call receiving phone calls manipulating images taking credit card payments lived at Piazza San Felice 4 walk to Accademia down old Florence past Pitti bridge Orsanmichele Duomo Baptistery Via Ricasoli to Piazza San Marco saw Florence at street level every condition empty winter evenings to sweltering summer days streets packed with tourists. paint people like still lives they willing portrait apex of still life painting long sitting pained expressions. Interleaf smart people built impressive technology crushed by Moore's Law. 1990s exponential growth in power of commodityIntel processors rolled up high-end special-purpose hardware software companies like bulldozer signature style seekers at RISD weren't specifically mercenary. In art world money and coolness coupled Anything expensive seen as cool equally expensive. apartment wasn't rent-controlled but rent-stabilized refinement New Yorkers know care about cheap less than half market price Most software can launch soon done. online store builder hosting stores if users yet fact painfully obvious before launch publicly had to launch privately recruiting initial users decent-looking stores code editor in Viaweb for users define page styles editing Lisp expressions't app editor code ran when merchants' sites generated not shoppers visited first instance of familiar experience comments full of angry people. claim Lisp better than other languages? Turing complete? responses to essays sorry feel for me not exaggerating always been like this since beginning comes with territory essay must tell readers things don't know some people dislike told such things People put stuff on internet in 90s not same as publishing online Publishing online online version as primary version[12] general lesson experience with Y Combinator teaches: Customs constrain after disappeared Customary VC practice like customs publishing essays based on real constraints Startups expensive start proportionally rare. Now cheap common VCs' customs reflected old world customs writing essays reflected constraints print era implies people independent-minded less influenced by custom advantage in fields affected by rapid change customs likely obsolete). can't always predict fields affected by rapid change software venture capital will be but predicted essay writing? [13] Y Combinator not original name. first called Cambridge Seed. didn't want regional name renamed after Y combinator. picked orange as color warmest no VC used it. In 2005 VCs used staid colors like maroon navy blue forest green appeal to LPs not founders. YC logo inside joke: Viaweb logo white V on red circle made YC logo white Y on orange square looked salvage plans Lisp. Lisp interesting for not just for association with AI decided to focus on Lisp decided to write book about Lisp hacking little knew about Lisp hacking when started writing booknothing like writing book about to help learn book, On Lisp published till 1993 wrote in grad school Computer Science uneasy alliance between theory and systems theory prove things systems build things I wanted to build things respect for theory suspicion more admirable but building seemed more exciting problem with systems work didn't last program wrote today good obsolete in couple decades People might mention software in footnotes no one use it seem feeble work Only people with history field realize time it been good. surplus Xerox Dandelions around computer lab Anyone wanted could briefly tempted slow by present standards what point? No one else wanted one off they went That happened to systems work wanted not just to build things things that last dissatisfied state in 1988 Rich Draves at CMU in grad school Carnegie Institute kid painting realized surprise wall something could make would last. Paintings didn't become obsolete Some best ones hundreds of years old could make a living doing Not easily as writing software but if industrious lived cheaply possible to make enough survive as artist could be independent have boss or need research funding liked looking at paintings.Could make them? no idea never imagined possible knew people made art didn't appear spontaneously different species lived long ago or were mysterious geniuses doing strange things in profiles Life magazine idea of art seemed almost miraculous started art classes at Harvard Grad students any department advisor Tom Cheatham easy going knew about strange classes never said in PhD program in computer science planning to be artist in love with Lisp hacking working at On Lisp like grad student working on multiple projects not my thesis didn't see way out didn't want to drop out of grad school how get out? remember friend Robert Morris kicked out of Cornell for writing internet worm of 1988 envious way to get out of grad school April 1990 crack appeared in wall professor Cheatham asked if far enough to graduate didn't have dissertation written decided to writing one in 5 weeks before deadline reusing parts On Lisp no delay read in few days.\" picked applications of continuations as topic invited to give talk at Lisp conference about used Lisp at Viaweb put postscript file of talk online on paulgraham created years before Viaweb never usedone day got 30,000 page views. What happened? referring urls showed someone posted it on Slashdot. thought there's audience. If I write something on web anyone can read it. obvious now surprising then. print era narrow channel to readers guarded by editors. only way to get audience for was published as book or newspaper magazine Now anyone could publish anything possible since 1993 not many people realized involved with building infrastructure web taken 8 years to realize took several years to understand implications. meant new generation of essays. print era channel for publishing essays small. Except for few anointed thinkers right New York only people allowed publish essays were specialists writing about specialties many essays never written no way to publish. Now could be I going to write them. worked on several different things turning point started publishing essays online knew'd always write essays online essays marginal medium at first Socially seem more like rants by nutjobs on GeoCities sites than compositions The New Yorker. encouraging instead of discouraging. well worked to work on things't prestigious. Still life least prestigious form of painting.Viaweb Y Combinator seemed lame when started still get glassy eye from strangers ask what I'm writing explain it's an essay publish on my web site Lisp prestigious intellectually like Latin seems hip. not unprestigious work good But drawn to work despite lack of prestige sign something real be discovered right motives Impure motives big danger for ambitious astray desire to impress people working on things't prestigious doesn't guarantee right track, guarantees not on common wrong one. years wrote essays about different topics O'Reilly reprinted collection book Hackers & Painters worked on spam filters did more painting dinners for friends every thursday night taught how to cook for groups bought another building in Cambridge former candy factory later porn studio), as office October 2003 big party at my house idea of friend Maria Daniels Three hosts invite friends to one party for every guest two thirds other guests people they didn't know but like. One didn know woman Jessica Livingston asked her out Jessica in charge of marketing at Boston investment bank bank thought understood startups met friends startup world surprised different reality"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import display, Markdown\n",
    "\n",
    "display(Markdown(results[0].text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query_engine1 = index.as_query_engine(\n",
    "    similarity_top_k=8, postprocessors=[compressor_llmlingua2]\n",
    ")\n",
    "\n",
    "response = query_engine1.query(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "During his time at Yale, the author was in a PhD program in computer science while also planning to become an artist. He took art classes at Harvard and was energetically working on multiple projects that were not related to his thesis."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(Markdown(str(response)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'f01289fa-87f1-4dcd-b0cc-6f44ae51629c': {'file_path': 'pg_essay.txt',\n",
       "  'file_name': 'pg_essay.txt',\n",
       "  'file_type': 'text/plain',\n",
       "  'file_size': 75042,\n",
       "  'creation_date': '2025-01-16',\n",
       "  'last_modified_date': '2025-01-16'},\n",
       " '2793a0b2-457e-4eae-b376-0c79906359ab': {'file_path': 'pg_essay.txt',\n",
       "  'file_name': 'pg_essay.txt',\n",
       "  'file_type': 'text/plain',\n",
       "  'file_size': 75042,\n",
       "  'creation_date': '2025-01-16',\n",
       "  'last_modified_date': '2025-01-16'},\n",
       " 'e5978a51-60fb-49b9-a743-1e9c45cf4f76': {'file_path': 'pg_essay.txt',\n",
       "  'file_name': 'pg_essay.txt',\n",
       "  'file_type': 'text/plain',\n",
       "  'file_size': 75042,\n",
       "  'creation_date': '2025-01-16',\n",
       "  'last_modified_date': '2025-01-16'},\n",
       " '6d9b4641-53ae-4a6d-989a-20cadd9415e5': {'file_path': 'pg_essay.txt',\n",
       "  'file_name': 'pg_essay.txt',\n",
       "  'file_type': 'text/plain',\n",
       "  'file_size': 75042,\n",
       "  'creation_date': '2025-01-16',\n",
       "  'last_modified_date': '2025-01-16'},\n",
       " '3bac5d02-c6a0-4e92-9e23-9286254fa682': {'file_path': 'pg_essay.txt',\n",
       "  'file_name': 'pg_essay.txt',\n",
       "  'file_type': 'text/plain',\n",
       "  'file_size': 75042,\n",
       "  'creation_date': '2025-01-16',\n",
       "  'last_modified_date': '2025-01-16'},\n",
       " '7748cd46-2588-4ec1-8220-2a8c85e684c7': {'file_path': 'pg_essay.txt',\n",
       "  'file_name': 'pg_essay.txt',\n",
       "  'file_type': 'text/plain',\n",
       "  'file_size': 75042,\n",
       "  'creation_date': '2025-01-16',\n",
       "  'last_modified_date': '2025-01-16'},\n",
       " 'dd56b836-1549-416e-824b-88989aca584c': {'file_path': 'pg_essay.txt',\n",
       "  'file_name': 'pg_essay.txt',\n",
       "  'file_type': 'text/plain',\n",
       "  'file_size': 75042,\n",
       "  'creation_date': '2025-01-16',\n",
       "  'last_modified_date': '2025-01-16'},\n",
       " '86d9d9e4-f480-43a0-9eb4-a898578aabe3': {'file_path': 'pg_essay.txt',\n",
       "  'file_name': 'pg_essay.txt',\n",
       "  'file_type': 'text/plain',\n",
       "  'file_size': 75042,\n",
       "  'creation_date': '2025-01-16',\n",
       "  'last_modified_date': '2025-01-16'}}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response.metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test llmlingua 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e429959792d6426189a15874b05f08ab",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "compressor_llmlingua1 = LongLLMLinguaPostprocessor(\n",
    "    device_map=\"mps\"  # Mac users rejoice!\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "We detected that you are passing `past_key_values` as a tuple and this is deprecated and will be removed in v4.43. Please use an appropriate `Cache` class (https://huggingface.co/docs/transformers/v4.41.3/en/internal/generation_utils#transformers.Cache)\n"
     ]
    }
   ],
   "source": [
    "results = compressor_llmlingua1._postprocess_nodes(\n",
    "    nodes, query_bundle=QueryBundle(query_str=query)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[NodeWithScore(node=TextNode(id_='4d757057-17ff-4bca-8846-e8aa907761ca', embedding=None, metadata={'f01289fa-87f1-4dcd-b0cc-6f44ae51629c': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, '2793a0b2-457e-4eae-b376-0c79906359ab': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, 'e5978a51-60fb-49b9-a743-1e9c45cf4f76': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, '6d9b4641-53ae-4a6d-989a-20cadd9415e5': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, '3bac5d02-c6a0-4e92-9e23-9286254fa682': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, '7748cd46-2588-4ec1-8220-2a8c85e684c7': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, 'dd56b836-1549-416e-824b-88989aca584c': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, '86d9d9e4-f480-43a0-9eb4-a898578aabe3': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, metadata_template='{key}: {value}', metadata_separator='\\n', text='I remember when my friend Robert Morris got of for writing the internetorm8, I was envious hed found a to get of.', mimetype='text/plain', start_char_idx=None, end_char_idx=None, metadata_seperator='\\n', text_template='{metadata_str}\\n\\n{content}'), score=None),\n",
       " NodeWithScore(node=TextNode(id_='8d6594c1-e649-47f3-a8eb-03b491c8086c', embedding=None, metadata={'f01289fa-87f1-4dcd-b0cc-6f44ae51629c': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, '2793a0b2-457e-4eae-b376-0c79906359ab': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, 'e5978a51-60fb-49b9-a743-1e9c45cf4f76': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, '6d9b4641-53ae-4a6d-989a-20cadd9415e5': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, '3bac5d02-c6a0-4e92-9e23-9286254fa682': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, '7748cd46-2588-4ec1-8220-2a8c85e684c7': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, 'dd56b836-1549-416e-824b-88989aca584c': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, '86d9d9e4-f480-43a0-9eb4-a898578aabe3': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, metadata_template='{key}: {value}', metadata_separator='\\n', text='Then day in90 a crack the wall. I ran into professor Cheam and he asked I was along toate that It have a word of my but in what been theest of thinking I to take a at writing one in the  weeks so remained before theline, reusing of Onisp could and I was to respond, with noible delay \" I soll give you something\\nI picked applications ofations topic. Inpect I have written about macros and embedded languages. Theres a whole world there thatsly. But I wanted to get of school and my rapidly writtenationed,\\nwhile I was applying to art I to two:D in the and the di, which, because was the school, Iined be.D I theia so to\\nId applied for the program, which meant in I to go to college. was not as sounds, because I was2 and art schools are of people of.D counted a transfer sore and said I to do the foundation that. The foundation means the classes that everyone take in fundamental subjects like,, and.', mimetype='text/plain', start_char_idx=None, end_char_idx=None, metadata_seperator='\\n', text_template='{metadata_str}\\n\\n{content}'), score=None),\n",
       " NodeWithScore(node=TextNode(id_='efa8fbba-f276-4249-b854-c6bd8d35bbdd', embedding=None, metadata={'f01289fa-87f1-4dcd-b0cc-6f44ae51629c': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, '2793a0b2-457e-4eae-b376-0c79906359ab': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, 'e5978a51-60fb-49b9-a743-1e9c45cf4f76': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, '6d9b4641-53ae-4a6d-989a-20cadd9415e5': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, '3bac5d02-c6a0-4e92-9e23-9286254fa682': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, '7748cd46-2588-4ec1-8220-2a8c85e684c7': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, 'dd56b836-1549-416e-824b-88989aca584c': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, '86d9d9e4-f480-43a0-9eb4-a898578aabe3': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, metadata_template='{key}: {value}', metadata_separator='\\n', text=\"T I a big surprise: from theia, which had been delayed because theyd to Cambridge England of Massachusetts, to the exam in Florence that. was now only. My niceady me leave my heric I had some money saved froming work done in school; was to last a year I. Now was\\nOnly stranieri) take this. Inpect it may been a ofcluding, because were manyieri the of studying in Florence that the Italian students would otherwise have been outed. I was in decent shape at painting drawing from theD foundation that I still don I to the. I remember that I answered the question by about, and I ced the intellectual level could to make the my limited. [\\nI'm only to25 and already are suchuous patterns. Here I, again to attend some institution in the of learning about someious and again to be. The students andulty in the theia were the people but they had since at anby the studentst require the teach anything and in return theulty wouldnt require to. And the time all involved would adhere out the of a  century at. We actually had of those little stes, with, that you in centuryings and a model sitting as possible without.cept anyone else painted her. The students timeatting or occasionally toitate in American\\nur model to live just from me She made a living from a of and making fakes for a. Shed copy anure painting, thend take the and m it. [3\\n was student theia I started painting still lives in myroom at. These were, because the was and I them onver of canvas, which afford.ting lives is painting because the subject, as, can.I kept on YC till, to help get that ofups through Day, then I out pretty.I still toi and to newups on but that\\nWhat? Rtms advicet included about I to something, I decidedd paint I to see how get I really. So the day I stopped working on I painting I wasy and into but was at least completely. [18\\nI spent the of 14 Id never been to work so I got to be I. Not good enough. Then in, right of I of. Up tilld been curious how the painting but suddenly finishing one. stopped anded my andt. So farI that sounds rather w. But a zero. you can choose to, choose a project thats not the best one (or least a) for, thens getting of. And at 0 there was some opportunity cost to aroundI started writingays, and a of ones I a couple that about. Then in March1 I started onisp\\nThe about that its a language by writing interpreter in. Itt originally language the sense was a formal, an the. you to write language in, what the minimum ofdefined? Theisp thatthyed, or moreately, is an to. [9\", mimetype='text/plain', start_char_idx=None, end_char_idx=None, metadata_seperator='\\n', text_template='{metadata_str}\\n\\n{content}'), score=None),\n",
       " NodeWithScore(node=TextNode(id_='5926969d-9925-4abf-bd50-b92428965653', embedding=None, metadata={'f01289fa-87f1-4dcd-b0cc-6f44ae51629c': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, '2793a0b2-457e-4eae-b376-0c79906359ab': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, 'e5978a51-60fb-49b9-a743-1e9c45cf4f76': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, '6d9b4641-53ae-4a6d-989a-20cadd9415e5': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, '3bac5d02-c6a0-4e92-9e23-9286254fa682': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, '7748cd46-2588-4ec1-8220-2a8c85e684c7': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, 'dd56b836-1549-416e-824b-88989aca584c': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, '86d9d9e4-f480-43a0-9eb4-a898578aabe3': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, metadata_template='{key}: {value}', metadata_separator='\\n', text='thyt realize thisisp could be to computers till his student Russell. Russell translateds into and pointisp started be language the. But its a computation a and that languages. was thated me in,t understand why\\ns0isp did nothing than interpret was missing of thingsd. So these added and, theyt defined usings original. That beenible time.thy tested his interpreter by hand- the of programs. But was already getting the ofters could test way —, a in thatthy had. test a have to it and computerst\\nNow,. you could continue usings approach you a complete. And as as every tos was a discoverednesserving could, complete language that had this. Hard do to, but if it possible, not? I to. took , 215 to11. was I a defined, or been to keep itI wrote this newisp, called Bel in itself in. That sound a its an the of I to make. By of an eiouscks make something close to an interpreter in that could. Not toI to ban from writingays during,. In I spent and I to Ily. was veryaging, because their were quite and the good founders were very. you were to learn the most could aboutups in the yout have\\n were the. Dis between, out when, with who m the I hard the was by something Kevinale said about \" one the.\" He it both was the that wanted be so if worked the\\n, when wasing California for, Morris did something he me un I can only remember him. day at Via, was over double a he that for to. was what for. I remember his words should sureator last\\n but should. This seemed was But if thing r thantm So this. was that on myC be the last I was only taking up my. It had alreadyaten wasatingays. Either was my life I to leave. And itt I\\n the2 my and the aot by The stroke her was, she and I to fly to her. was toI askedica be shet so we decidedd toruit Sam. We to and and we to a complete. point been the original LLC we wanted to it So Samd him. would and become\\n, startup make I in We over the. the1 I left runningC more to, could the was my,\\n on. We knew hardIC, I. still to and toups on thatWhat advice about I something I get day was and it get into was completely spent the4 so be. I\\n0\\nBefore college the on, I writeays wrote what writers write and were had, characters with Iined first I were the1 school for  school districts401 the of our and my friend Rich got was a miniir these machines — CPU, — sitting floor underThe was. type then to The would be print thely was the. Andpects only to programs was data I any only to do thatt any like calculate interesting I not I they have. My the learned programs to when. without- a technical the data managers\\nWith computer running just a and then. aer was by remember him programs\\n of The but was. to my a that write. was in2 lot\\n It which more seemed, the things mere. I the fields of for these. that were that fields feltI I kept to9 but were things a by which a showed It tried but was drawn seemed have. All wasU.The,, thet at was the and running Via. to to my usual of programming butigue withsly aged andim Clara gradually felt dis like at\\n had of. was so be but I first beendd forgotten My been and mensts had get could I\\n I atahoo him the of. was was. My were million leaving it take the was was.aving startup to him an and yet also, I was to I started. already years getting Now I to are leaving after,., It any. know ind comp by house but I few, where you understand about still, se up tomb. Id York at and people to,\\n I was as I were doors. of, was and I past order was.ting I life painting the then, then the a second same objectswhich hopefully\\n I to. actually. myself and the? A by to I.\\n was from that for people server then?] run of the that use by call: making and receiving,, taking etc I at, my theia went the of the past, San I at from empty[] can course paint peoplere That is the the long produce] Inter was of many had and built yet by the ofityors up like[ The signature styleers atDt specifically and.ing expensive become] thet aement only was,] Most software can as it But is an storere the, that So launch to ofing had[] a code in for to editing expressions an app code the were when first familiar and was I the of. I that better than? all? who the to how not I It.ay must tell and people] People put of internets but it you the a general that our withator: Customs to the.ary had onces publishing ess, been on real. had been more andally they thes\\' still, as the\\nich turn areed. less in fields by\\n:. and] the called, case someone in Sil, so we renamed ourselves after theest theI picked color the because noC5 used like to to logo the Via. I could my was was its, decided to on.isps, in\\n an two and people had of — a more but seemedt. ob a coupleades at. People might mention your in And seem veryble.\\n someplusx at. one was by standards;? so. to systemsI wanted this went visit Rich at. the Instituted. While a realized that, something could wouldt.And moreover as really and lived it And be get liked. never knew didnt just appearously was They either lived wereiouses in in make to that verb almost\\nThat., Tom very. he the a, yet, also withisp and working On, was working way of this I to of but I got of internetorm found a in Che was to that theest thinking weeks Onisp no \" so to of. was a about I a, on created before usingweb In. What The referring that.\\n., That the a to, byce only get an for published\\nThis been with and a. It meant of. the the been. few officially who people were to publish. wasve but a I started.I that a.d likeants by their the andet But by to thating mostuous how thatt. Still life the least. Via and seemedame them Im. Even thoughious in, seems as thatious are work its sign something of. If to So whiletioustre least the most common\\n wrote. Oilly calledters on I have every which I bought factory, twas,3 a friend Maria the. Three separate hosts would all invite their friends to one party. So for every guest, two thirds of the other guests would be people they didn\\'t know but would probably like. One of the guests was someone I didn\\'t know but would turn out to like a lot: a woman called Jessica Livingston. A couple days later I asked her out.', mimetype='text/plain', start_char_idx=None, end_char_idx=None, metadata_seperator='\\n', text_template='{metadata_str}\\n\\n{content}'), score=None),\n",
       " NodeWithScore(node=TextNode(id_='f13d07f8-4ba0-4f84-863a-3be74e61639e', embedding=None, metadata={'f01289fa-87f1-4dcd-b0cc-6f44ae51629c': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, '2793a0b2-457e-4eae-b376-0c79906359ab': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, 'e5978a51-60fb-49b9-a743-1e9c45cf4f76': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, '6d9b4641-53ae-4a6d-989a-20cadd9415e5': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, '3bac5d02-c6a0-4e92-9e23-9286254fa682': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, '7748cd46-2588-4ec1-8220-2a8c85e684c7': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, 'dd56b836-1549-416e-824b-88989aca584c': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}, '86d9d9e4-f480-43a0-9eb4-a898578aabe3': {'file_path': 'pg_essay.txt', 'file_name': 'pg_essay.txt', 'file_type': 'text/plain', 'file_size': 75042, 'creation_date': '2025-01-16', 'last_modified_date': '2025-01-16'}}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, metadata_template='{key}: {value}', metadata_separator='\\n', text='Jessica was in charge of marketing at a Boston investment bank. This bank thought it understood startups, but over the next year, as she met friends of mine from the startup world, she was surprised how different reality was.', mimetype='text/plain', start_char_idx=None, end_char_idx=None, metadata_seperator='\\n', text_template='{metadata_str}\\n\\n{content}'), score=None)]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "During his time at Yale, the author was in a PhD program in computer science while also planning to become an artist. He took art classes at Harvard and was working on writing a book about Lisp hacking."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "query_engine_llmlingua1 = index.as_query_engine(\n",
    "    similarity_top_k=8, postprocessors=[compressor_llmlingua1]\n",
    ")\n",
    "\n",
    "response = query_engine_llmlingua1.query(query)\n",
    "\n",
    "display(Markdown(str(response)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'f01289fa-87f1-4dcd-b0cc-6f44ae51629c': {'file_path': 'pg_essay.txt',\n",
       "  'file_name': 'pg_essay.txt',\n",
       "  'file_type': 'text/plain',\n",
       "  'file_size': 75042,\n",
       "  'creation_date': '2025-01-16',\n",
       "  'last_modified_date': '2025-01-16'},\n",
       " '2793a0b2-457e-4eae-b376-0c79906359ab': {'file_path': 'pg_essay.txt',\n",
       "  'file_name': 'pg_essay.txt',\n",
       "  'file_type': 'text/plain',\n",
       "  'file_size': 75042,\n",
       "  'creation_date': '2025-01-16',\n",
       "  'last_modified_date': '2025-01-16'},\n",
       " 'e5978a51-60fb-49b9-a743-1e9c45cf4f76': {'file_path': 'pg_essay.txt',\n",
       "  'file_name': 'pg_essay.txt',\n",
       "  'file_type': 'text/plain',\n",
       "  'file_size': 75042,\n",
       "  'creation_date': '2025-01-16',\n",
       "  'last_modified_date': '2025-01-16'},\n",
       " '6d9b4641-53ae-4a6d-989a-20cadd9415e5': {'file_path': 'pg_essay.txt',\n",
       "  'file_name': 'pg_essay.txt',\n",
       "  'file_type': 'text/plain',\n",
       "  'file_size': 75042,\n",
       "  'creation_date': '2025-01-16',\n",
       "  'last_modified_date': '2025-01-16'},\n",
       " '3bac5d02-c6a0-4e92-9e23-9286254fa682': {'file_path': 'pg_essay.txt',\n",
       "  'file_name': 'pg_essay.txt',\n",
       "  'file_type': 'text/plain',\n",
       "  'file_size': 75042,\n",
       "  'creation_date': '2025-01-16',\n",
       "  'last_modified_date': '2025-01-16'},\n",
       " '7748cd46-2588-4ec1-8220-2a8c85e684c7': {'file_path': 'pg_essay.txt',\n",
       "  'file_name': 'pg_essay.txt',\n",
       "  'file_type': 'text/plain',\n",
       "  'file_size': 75042,\n",
       "  'creation_date': '2025-01-16',\n",
       "  'last_modified_date': '2025-01-16'},\n",
       " 'dd56b836-1549-416e-824b-88989aca584c': {'file_path': 'pg_essay.txt',\n",
       "  'file_name': 'pg_essay.txt',\n",
       "  'file_type': 'text/plain',\n",
       "  'file_size': 75042,\n",
       "  'creation_date': '2025-01-16',\n",
       "  'last_modified_date': '2025-01-16'},\n",
       " '86d9d9e4-f480-43a0-9eb4-a898578aabe3': {'file_path': 'pg_essay.txt',\n",
       "  'file_name': 'pg_essay.txt',\n",
       "  'file_type': 'text/plain',\n",
       "  'file_size': 75042,\n",
       "  'creation_date': '2025-01-16',\n",
       "  'last_modified_date': '2025-01-16'}}"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response.metadata"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llamaindex",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
